MINOR: add wait_for_assigned_partitions to console-consumer #8192
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brianbushree
force-pushed
the
throttle-test-fix
branch
from
3a3d718 to
26588bf
Compare
brianbushree
commented
Feb 28, 2020
| @@ -273,8 +276,25 @@ def _worker(self, idx, node): | |||
| with self.lock: | |||
| self.read_jmx_output(idx, node) | |||
|
|
|||
| def _wait_until_partitions_assigned(self, node, timeout_sec=60): | |||
| if self.jmx_object_names is not None: | |||
| raise Exception("'wait_until_partitions_assigned' is not supported while using 'jmx_object_names'/'jmx_attributes'") | |||
There was a problem hiding this comment.
this is not ideal but requires more refactoring to be able support multiple JmxTools running at once
|
just confirmed that this doesn't break |
brianbushree
changed the title
brianbushree
changed the title
bbejeck
reviewed
Feb 28, 2020
There was a problem hiding this comment.
This LGTM, but since I'm not too familiar, I'd like to get a second pair of eyes on this.
one of @hachikuji, @mattwong949, @gardnervickers
|
hit one timeout with |
|
Ran on branch builder (the 2.4 branch PR) with 5 repeats, all tests passed re-running the trunk PR version. |
mattwong949
approved these changes
Feb 28, 2020
brianbushree
commented
Feb 28, 2020
| jmx_tool.start_jmx_tool(self.idx(node), node) | ||
| jmx_tool.read_jmx_output(self.idx(node), node) | ||
| assigned_partitions_jmx_attr = "kafka.consumer:type=consumer-coordinator-metrics,client-id=%s:assigned-partitions" % self.client_id | ||
| wait_until(lambda: assigned_partitions_jmx_attr in jmx_tool.maximum_jmx_value, |
There was a problem hiding this comment.
we should be re-running read_jmx_output before checking this condition
|
they're passing for me with |
|
Ok to test |
|
previous branch builder test run failed. Re-running the test now. If the test passes, I'll merge this and cherry-pick back. |
|
2nd branch builder run passed. |
|
merged #8192 into trunk Thanks @brianbushree! |
bbejeck
pushed a commit
that referenced
this pull request
Mar 1, 2020
what/why the throttling_test was broken by this PR (#7785) since it depends on the consumer having partitions-assigned before starting the producer this PR provides the ability to wait for partitions to be assigned in the console consumer before considering it started. caveat this does not support starting up the JmxTool inside the console-consumer for custom metrics while using this wait_until_partitions_assigned flag since the code assumes one JmxTool running per node. I think a proper fix for this would be to make JmxTool its own standalone single-node service alternatives we could use the EndToEnd test suite which uses the verifiable producer/consumer under the hood but I found that there were more changes necessary to get this working unfortunately (specifically doesn't seem like this test suite plays nicely with the ProducerPerformanceService) Reviewers: Mathew Wong <[email protected]>, Bill Bejeck <bbejeck.com>
|
cherry-picked to 2.4 |
qq619618919
pushed a commit
to qq619618919/kafka
that referenced
this pull request
May 12, 2020
) what/why the throttling_test was broken by this PR (apache#7785) since it depends on the consumer having partitions-assigned before starting the producer this PR provides the ability to wait for partitions to be assigned in the console consumer before considering it started. caveat this does not support starting up the JmxTool inside the console-consumer for custom metrics while using this wait_until_partitions_assigned flag since the code assumes one JmxTool running per node. I think a proper fix for this would be to make JmxTool its own standalone single-node service alternatives we could use the EndToEnd test suite which uses the verifiable producer/consumer under the hood but I found that there were more changes necessary to get this working unfortunately (specifically doesn't seem like this test suite plays nicely with the ProducerPerformanceService) Reviewers: Mathew Wong <[email protected]>, Bill Bejeck <bbejeck.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
what/why
the
throttling_testwas broken by this PR (#7785) since it depends on the consumer having partitions-assigned before starting the producerthis PR provides the ability to wait for partitions to be assigned in the console consumer before considering it started.
caveat
this does not support starting up the JmxTool inside the console-consumer for custom metrics while using this
wait_until_partitions_assignedflag since the code assumes one JmxTool running per node.I think a proper fix for this would be to make
JmxToolits own standalone single-node servicealternatives
we could use the
EndToEndtest suite which uses the verifiable producer/consumer under the hood but I found that there were more changes necessary to get this working unfortunately (specifically doesn't seem like this test suite plays nicely with theProducerPerformanceService)Committer Checklist (excluded from commit message)